Skip to content

[FLINK-39251][Formats] Support Nanosecond Logical Types in Flink Avro #27770

Open
talatuyarer wants to merge 1 commit intoapache:masterfrom
talatuyarer:avro-nanosecond-support
Open

[FLINK-39251][Formats] Support Nanosecond Logical Types in Flink Avro #27770
talatuyarer wants to merge 1 commit intoapache:masterfrom
talatuyarer:avro-nanosecond-support

Conversation

@talatuyarer
Copy link

What is the purpose of the change

Currently, the Flink Avro format support for TIMESTAMP and TIMESTAMP_WITH_LOCAL_TIME_ZONE is capped at microsecond precision (6).
Standard Apache Avro (up to 1.11) only defines logical types for timestamp-millis and timestamp-micros.

When Flink tables utilize nanosecond precision (TIMESTAMP(9)), there is no native Avro logical type mapping, leading to either fallback behaviour (treating as raw BIGINT) or unsupported type exceptions during automatic schema conversion.

This pull request adds support for nanosecond-precision logical types (timestamp-nanos and local-timestamp-nanos) in the Flink Avro format. It also upgrades Avro to version 1.12.1 and enables decimal logical types handling for improved compatibility and numeric fidelity.

Specifically:

  • Supports mapping Flink's TIMESTAMP(9) and TIMESTAMP_WITH_LOCAL_TIME_ZONE(9) back-and-forth with Avro's timestamp-nanos and local-timestamp-nanos logical types.
  • Enables high-precision temporal calculations without losing precision at microsecond or millisecond boundaries.

Brief change log

  • AvroToRowDataConverters.java:
    • Replaced the static timestamp converter with a precision-aware createTimestampConverter(int precision) that handles millisecond, microsecond, and nanosecond cases into TimestampData.
  • RowDataToAvroConverters.java:
    • Updated conversion for TIMESTAMP_WITHOUT_TIME_ZONE and TIMESTAMP_WITH_LOCAL_TIME_ZONE to respect higher precision values up to 9 (nanoseconds) when packing epoch values.
  • AvroSchemaConverter.java:
    • Mapped LogicalTypes.timestampNanos() and LogicalTypes.localTimestampNanos() schemas into equivalent Flink SQL styles (TIMESTAMP(9) / TIMESTAMP_WITH_LOCAL_TIME_ZONE(9)).
    • Updated conversion from Flink data types to Avro schema to support precision up to 9 for timestamp types.
  • pom.xml:
    • Upgraded Avro dependency to 1.12.1.

Verifying this change

This change added tests and can be verified as follows:

  • Updated unit tests within AvroSchemaConverterTest.java (e.g., asserting fields like type_timestamp_nanos, type_local_timestamp_nanos map correctly).
  • Ran pre-existing Avro suite (AvroOutputFormatTest, AvroTypeExtractionTest) to ensure backwards compatibility with upgraded baseline versions.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): yes (Upgraded Avro to 1.12.1)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes (Adjusts converter inner-loops, standard for type feature additions)
  • Anything that affects deployment or recovery: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? not applicable (Automatic type mapping addition, validated by updated test cases)

@talatuyarer
Copy link
Author

@mxm @gyfora Could you review this pr ?

@flinkbot
Copy link
Collaborator

flinkbot commented Mar 14, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

+ "with precision: "
+ precision
+ ", it only supports precision less than 6.");
+ ", it only supports precision less than 9.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It supports 9 as well I think.

@davidradl
Copy link
Contributor

@talatuyarer so we support with a precision of 4 but is gets treated as 6?

Copy link
Contributor

@MartijnVisser MartijnVisser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be very much in favor of splitting off the Avro dependency upgrade into its own ticket, instead of including it here.

Copy link
Contributor

@raminqaf raminqaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. I have a related PR 27757 to your changes for supporting precisions of 0-9 for the TO_TIMESTAMP_LTZ function. One thing I noticed in your changes and would like to raise awareness is:
Non-standard precisions (1, 2, 4, 5, 7, 8) are silently mapped to the next Avro-supported bucket (millis/micros/nanos) without any validation or warning. A TIMESTAMP(5) gets mapped to timestamp-micros but the converter rounds to micros, potentially losing the 5th fractional digit. Should we either (a) throw a validation error for unsupported precisions, or (b) explicitly document/warn about the truncation?

@gaborgsomogyi
Copy link
Contributor

+1 to separate version bump from this feature. Additionally it would be good to add test cases for all precisions. Proper mapping is essential here.

@gaborgsomogyi
Copy link
Contributor

Seems like the LicenseChecker blown up, can you plz fix it to see test run?

@github-actions github-actions bot added the community-reviewed PR has been reviewed by the community. label Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants